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ABSTRACT . 

The costs incurred by state education agencies 
(SEA's) for evaluation' activities are looked at in this study, which 
also examines \he possible role that the use of management consulting 
might play in reducing these costs. It is estimated that SEA 
evaluation activities in the U.S. today cost about $32 taillicyi and 
that the use of management consulting' offers potential savings oflF 
about $2.9 million. The study compares management consulting, 
technical consulting, evaluation, and research activities and 
explains how each is different. Although management consulting and 
evaluation activities may overlap, one distinct difference is that 
management consulting is client centered regarding decision-making 
whereas evaluation activities emphasize a critical record of program 
activities or outcomes. The study Concludes with a number of 
recommendations for further research, including a refinement and 
cdnfirmation of the figures presented here, and studies to show how 
actual savings might be achieved" in practice. (Author/JM) 



( 



************** 
* 



************************ ** ****** *********************** 

* 
* 



Reproductions supplied I$y EDRS are. the best that can be made 
\. s ^ from the or iginal^'document . 1 



*********************************************************************** 



ERJC 



v 



U S DEPARTMENT Of EDUCATION 

NAMiiNAi iNbTiiurt ui- tuuLATiuN PERMISSION TO REPRODUCE THIS 



MATERIAL HAS BEEN GRANTED BY 

TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER 1ERIC1 




i 



ERIC 



:f 





Nor69 PILOT FIELD STUDY OF SEA 
EVAl.UAfiPH COSTS 



. ' Joiiathan StanHeld 




Northwttt Regional Educational Laboratory 




.PocH«nd,:Oregon 97204 J 
Tetepbone (503) 248-6800 



' ( 



■ «: „♦* 



*2 



No. 69 PILOT FIELD StUDY OF SEA 
EVALUATION COSTS 



JONATHAN STANFIELD 



Private Consultant 
Kirkland, Washington 



• — ^» 



March 1982 



Nick- L. Smith, Director 
Research on Evaluation Prograin 
Northwest Regional Educational Laboratory 
300 S..&7. Sixth Avenue, Portland, Oregon v<}7204 



1 



. /• 



Published by the Northwest Regional Educational Laboratory, a private 
nonprofit corporation. The work upon which' this publication iff based 
was performed, in part, pursuant to Contract No. 400-80-0105 pf the 
National Institute of Education. It does not, however, necessarily \ 
reflect the views of that agency. , 

The information presented in this publication does not necessarily 
reflect the opinions of the Northwest Regional Educational Laboratory 
and no endorsement, should be inferred. 



ERiC 



"i 



v 



PREFACE 



The Research on Evaluation Program is a Northwest Regional 
Educational Laboratory project of research, development, testing, 
and 'training designed to create new evaluation methodologies for 
use in education. This document is. one of a series of papers and 
reports produced lpy program staff, visiting scholars, adjunct 
scholars, and project collaborators — all members of a cooperative 
network of colleagues working on the development of new 
methodologie s . 

What are the costs associated with evaluation at the state 
level? Can management consulting techniques be used to reduce 
some of. these costs? After summarizing the costs of evaluation 
today (approximately $32 million at the state level), Dr. 
Stanf ield reviews the ways in which management consulting 
techniques could replace standard evaluation approaches or save 
as much as $2.9 million. Important distinctions in considering 
evaluation costs are also included in this pilot study report. 

Nick L. Smith, Editor 
Paper and Report Series 
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SUMMARY 

This pilot study examines SEA evaluation costs and 
the possible role that management consulting might 
play in reducing them. 

It is estimated that SEA evaluation, in the US today, 
costs about $32 million, and that management consulting 
offers potential savings 'of about $2.9 million in this 
figure. ' In addition, even greater savings may also 
T)e possible at the federal level. 

The study compares and contrasts management consulting, 
technical consulting, evaluation, and research. 
Management consulting and evaluation have areas of 
overlap, *>and also areas of distinctive' difference , 
the main one of which is in the client-centered 
emphasis of management consulting on advice for 
decision making, in contrast to evaluation's emphasis 
on a critical record of program activities and outcomes 

The study lists a number of activities that further 
"research might follow to refine and confirm these 
figures, and^to ^how how actual savings might be 
achieved iri practice. 

1 
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1. INTRODUCTION 



This is the final report -on the study: Pilot Field 

Study of SEA Evaluation Costs , carried out by the 

> 

consultant for the Northwest Regional Educational 
Laboratory (NWREL) , under the direction of Dr Nick 
Smith, Director, Research on Evaluation Program,, 



Recant Program activity has emphasized three major 
aspects of- evaluation, nainely, methodology , impact 
and practice * Methodology has been explored, for. 
example, in the metaphor series of papers, in which 
potential new approaches to evaluation are Sought 
among metaphors derived from practices in other fields, 
©•g-f operations, ra search (Pag^, 1979) $ investigative- 
journalism (Guba, 1979) • The question of the impact 
of evaluation has been explored in a number of papers 
on SEA evaluation and policy, which are summarized by- 
Smith (1980). These papers are also concerned with 
describing practice, that is, what actually goes on 
in SEA evaluation 'units , as an a^i establishing 
a more, realistic framework for dis-cussion and research. 
->A previous paper by the, present consultant builds upon 
this work in reviewing^SEA evaluation practice from a 
management consultant f s point" of vikw (Stanfield, 1981), 

The present study is part of the Program's move to 
< open up a neto area of research, dealing with th£ costs 
of evaluation, a subject that is not only complementary 
ijo^the Program's earlier work, but especially timely, 
0 in the light of current ecQnomic conditions . ^As a 
pilot study, therefore, its goals are threefold t to 
develop an initial approach to the analysis of SEA 
evaluation costs, to develop initial .quantitative 
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estimates of these costs, and to suggest directions 
for further research in this area. 

Ih addition, the si;udy considers the possible 
application of management consulting techniques in 
reducing SEA evaiifetiori costs. ^This is done because 
the Program is aware ^of some highly cost-effective 
exalnples of evaluation carried out by management 
consultants ih other contexts, and wishes to explore 
whether or not, and under what conditions, their 
^techniques might' be applicable to SELfV evaluation. 

'costs of SEA evaluation, like the coists of other 
internal services such as computing centers, can be 
viewed from two points of view. Ftpm the inside, or 

'supply point of view, we loQk alt the 1 cost of s operating 
the evaluation unit and-jproviding service to its clients, 
how these costs are made up, and means of reducing them 
for a particular service lgvel,. Ppom the .outside, or 
demand point of view, we look at the pricjs ' paid for 
obtaining certain services in the context of the value 
of the programs, and the ijmportance of the policy - 
issues*, 'to which they reiata. This naturally leads 
onto the broader question of the optimal expenditure 
on evaluation, rather than the solely minimizing 
approach of reducing costs. ' (This optimal question, 
which is' difficult and largely unresolved, is discu^ged 
briefly in Appendix A.) " * 

Tlte stijdy has sought to. minimize the burden placed upon 
field Veapondents at this stage, and consequently^ x 
field .enquiries have been brief and highly focused, 
Valual&e help, both in iftatters of data and of judgment, 
been given by SEA evaluation units io California, 



Montana and Washington, two school district evaluation 
units in Washington, and the National Center for 
Educational Statistics. A modest literature search 
on evaluation costs using* ERIQ indexes was not very 
productive. Two sources, however, that do deserve 
to be mentioned here are Boruch and Cordray (1980) 
and California State Department of Education (1978). 
The first summarizes -a great deal of descriptive 
material orj program evaluation at various levels, 
and the second outlines in detail the budget structure 
of one of^Jbhe nation's leading evaluation units. 

J. 

The report, itself, is presented ,in a number 
sections, dealing withi 

- the costs of SEA evaluation , i.e., how much 
money is spent on evaluation, where does it 
come frdm, what kinds of tasks are carried 
put, at* what cost, and so on; 

\ management ^consulting , i .-e . , what is 
distinctive about , management consulting 
in relation to evaluation, and - what ,are 
• the conditions under which it could be 
applied ; • 

\ > \* 

' - discussion of the applicability of 

management consulting (and other 

techniques) to SEA* evaluation and an \ 

* * estimate of the potential' sayings that 

might be achieved through such 

application? and 




x t - recommendations on further research 

based upon the experience of the pilot. 

Appendix A discusses experience with management support 
systems in other contexits to see if_this might throw 
any light on the costs of evaluation as a support tool 
-for educational management.. Appendix B lists ■ 
reference^ used in «the stutfy. 

The consultant would like to tha^nk the Program Director, 
Dr Nick Smith, for helpful discussions' on earlier 
versions of this report. The views expressed fi however , 
remain those of the consultant, and are not necessarily ■ 
those of the Program. 



SEA EVALUATION COSTS . , • 

« 

SEAs differ considerably in the way in whiah they 
approach- the management of a state's .educational 
enterprise. Within SEAs, evaluation units- differ 
similarly, not least in, size and sophistication.. 
Evaluation, itself/ as practiced in SEAs, is a multi- 
faceted .activity often difficult to raduce to a <* 
compact set oT headings. Consequently, in any 
attempt to quantify and analyze the costs associated 
with SEA evaluation there is bound to be a considerable 
degree of simplification. This will, of course, be 
familiar to evgluators concerned with the limitations 
of summativfe evaluation. Nevertheless, it is 
believed that the estimates presented below are 

a 

sufficiently re'alistic to serve as a useful first 
step in describing SEA evaluation costs' and in guiding 
further research beyond this pilot study*. The^y should 
not, however, be taken out of context. ' 

In deriving these e4±imates it has been necessary^ tq 
be pragmatic, utilising sources as available. * In 
some cases there are reliable and relevant printed 1 
sources, e.g. /from the -National Center for Educational 
Statistics. - In other cases, the three SEA evaluation 
units contacted (California, Montana, Washington), as 
well v as a number of other respondents, have provided 
information and estimates from which the consultary* 
has iften derived the figures below. To save repeating 
their 'names , these three state evaluation units will 
be referred to as the respondent states, in what 
follows. * 
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Estimated of SEA evaluation. costs are now discussed, 

under the headings: 

> 

j 

- costs of education; 

•A 

- costs of s ^As ; 

- costs of SEA evaluation? 

* - SEA evaluation costs breakdown; and 

i - individual evaluation tasks and costs. 

i * 

Unless otherwis^ stated, they are for 1980-81. x 



2.1 Costs of Education , 

The US currently* spends about $180 billion annually 
on education $f all kinds, of which, about $93 billion 
is spent on the segment of interest to this pilot 
study, namely, public elementary and secondary 
education. This figure has been growing at about 
9/^ per year- over the last* decade?, but is likely to 
slow down now. 



Funding is split between federal, state and local 
sources in the approximate ratio 9%t^S%l^%* This 
v ratio *ias remained fairly constant over the last 
decade, with a slight trend towards more state 
funding and away from local funding. 
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Expenditure varies considerably bjr v state, but when 
size is discounted, expenditure per capita is broadly 
comparable . 
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There are over ^0 million pupils s^ved by over 16,000 
local school* districts . 



S£j£ve 



Derivation . These figures are taken from the Digeat 
of Education .Statistics, 1979 edition, quoting the 
National Center for Educational Statistics. Totals * 
are updated by the 9% growth rate for 1969-79. The 
figure for public elementary and secondary education 
is obtained by discounting the figure given in the 
Digest by 10%, since the latter figure includes some 
other schools, e.g., residential schools for 
exceptional children, federal schools for Indians, 
federally operated elementary and secondary schools 
on posts, and sub-collegiatd departments of colleges. 



2 Costs of S 



Total costs qf SEAs of relevance to this study, i.e., 
cpncerned with local public education, are estimated 
to be about $700 million annually. Thus it costs 
about three quarters of one percent of the total 
education ^budget to manage it at a state level." *** 
There are^ • of . course, additional -management and r 
. supervisory costs at the local level (county, district, 
and individual school), but these are outside, the scope 

of the present study. 

• *. 

SEA costs vary considerably from state to state, most 

obviously as a function of size of state . 

♦ * 

Derivation . This figure is derived from a' recent 
survey of SEA expenditure, carried out by 'the National 
Center for Educational Statistics, not yet published. 



It should be noted that gross costs of SjSAs total over 
$1300 million annually, 'but include a variety of items 
that are not relevant to this study, such as state 
libraries-, public health services, school bus drivers, 
special education schools, vocational rehabilitation, 
and' so on* *^ 



Costs of SEA Evaluation 

^valuation of elementary and secondary education is 
carried out at federal, state and local levels. 
Consequently, before examining the cost of evaluation 
at the state level in detail, it seems useful to try 
to give some sense of the-r<Aative proportion between 
levels . Estimates of current expenditure on evaluation 
at the 'different levels aret 

"federal. $25 million 

' state * 32 

local 96 

$153 million 

Derivation * The federal figure is based upon data 
given by Boruch and Cordray (1980)^«r-^ontract 
expenditure on evaluation tjy the Office of Evaluation 
and Dissemination of the US Depkrtment of 'Education, 
plus 10?5 for internal activities, e.g., confleract 
servicing. This is almost certainly an underestimate 
of federal evaluation activities, perhaps by a sub- 
stantial margin. % 

The state level figure is based upon individual SEA 
evaluation costs for the 'respondent states, together 
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• with National Center for Educational Statistics figures 
(unpublished for planning and research at the state, 
level, a sojaewhat broader category. This figure was 
checked with staff ing- estimates for consistency (see 
^section 2*.^b bfelow) . It/€tppear^3_that this estimate 
is highly dependent upon how the- scope of evaluation 

, is defined (see section 2.^c below). 

The lQca\ figure is based upon erflftiiries with two LEAs 
(in Washington), together with the judgment of the 

respondent states on the ratio of LEA to SEA expenditure 

> j 

on evaluation. 5?he ratio of three shown is considered v 
a reasonable firsts estimate. 

\ 

The 'figures "on SEA and LEA evaluation appear broadly 
consistent with those of Frankel et al (1979), but 
are not explicitly related due to different scope 
definitions. Frankel et al contacted over 6000 
organizations in 1976-78 in a major study of national 
acti^i1;ies, staffing and costs in educational research, 
development^ dissemination and evaluation (RDD&E). 

However, the funding; for this evaluation is somewhat 
different. The reason for this lies in the difference 
between the original source of the funds, and where the 
funds are finally spent. For example, an, SEA 
evaluation budget includes money from federal as well 
as state, sources. In turn, an LEA budget includes 
money from federal and state as well as local sources. 
Thus*, federal funding on evaluation not oijly includes 
'federal expenditure , but also portions of state and 
looal expenditures . -The same adjustment is. made for 
- state and local levels. Estimates for the funding 
of "evaluation of elementary and secondary education are: 



9 

ERLC 



- 9 - 



lb 



federal 



$58 million 



local 



state 




$152 million 



^he apparent discrepancy in totals is due to rounding. 



erivation T^ The percentage of SEA evaluation budgets 
supplied £t*om federal funds is -estimated at 50?* based 
upon the experience oZ the respondent states. For 
PEA breakdown, Boruch and Cordray (1980) quote a UCLA 
tudy finding that 18% o£ LEA evaluation is federally 
unded. The balance is apportioned to state and local 
ources in proportion to the source of total education 
funds from these two sources (see section above Jj. 

When these figures are put alongside the estimates 
above for the funding of elementary and secondary 
education, we can see/that, for every dollar of , 
education funded, the federal government spends about 
$.007 on evaluation", or a little over one half of one 
percent. The comparable figures for state and f&cal 
levels are around $.001 ($.0013 and $.0009 respectively ) 
One way of interpreting* these figures is to say that 
the federal government is about seven times more 
concerned with the results of its funded programs than 
state or local government. On the other hand, this 
may simply reflect the fact that federal funding is 
often for more demanding or sensitive student 
constituencies.. Anothe^ factor in the small state 
and local ratios is- that evaluation at these levels is 
often not provided for explicitly at the time of funding 
and consequently there is a temptation to reduce 
expenditure on evaluation in favor of the main programs 






when^the time comes to *spend the money* These ^figures 
may be contrasted with the suggestion that between' . V 
tl^ee and five percervt should be spent on the , Iv^lu&'tiofl. 
of programs of interest (Alkin, 1980,) . * ^ • 

Some local expenditure takes place at the building 
level, but this has not been estimated, separately. , , 

i* 

As might be expected, ,there/is considerable^ctiff erence , 
in the actual s£ze of SEA,/ evaluation units (Caulley- 
• c and Smith, 1978)^ and consequently £n their budgets, 
which range from under ope hundred thousand dollars 
to over five million (Franke$ et ai, ^979 ) • 



2 A SEA Evaluation Costs Breakdown 

In this section we will examine SEA evaluation costs < 
more closel^ - the $32 million figure estimated above. 

(a) Outside* contracting. Practice regarding outside' 
contracting differs widely~b6-flween states 
(Caulley and Smith, 1978).; An estimate of the 
* average proportion contracted out is 35$* based ^ - 
upon the experience of the respondent states i' 
Thus, out?ide contracting totals $11.2 million, 
leaving $20.8 million for internal costs. 

Contracts matjy be for production items, as .in , 
statewide testing programs/, for actual evaluations, 
dr for^other services. A Contracts may range from 
around one thousand^to almost one "million .dollars , 
again based upor/ the experience of the resjJfondent 
states. 



;- s • : > " . V ' ■ > 

(b) Internal cost breakdown * .Most internal costs 

are staff Salary co3ts, which often lump together 

professional and clerical, full and part-time, 

with the remainder made up. of serviSe, supply and 

travel costs. Bas^d upon the experience of the 

respondent states, estimates fpr the breakdown of 

internal costs arei 
\ 

, salaries' , " $1 ?• 7- million • . 
services, etc. 3»1 15 



* $20.8 100^ ^~r^ f 



These * staff costs reflect an average of about 

$27,000 per capita, including fringe benefits. 

9t 

This corresponds to an SEA evaluator national * • * 
population (professional plus clerical) or about 
650. 

These figures\are' based upon the experience of the 
respondent states, cross checked with Caulley and 
Smith (1978) and Cronin (.1980). Caulley and Smith's ) 
figures, grossed up on a population t>asis to the 
whole country,! come to StJiout 50$ (in 19(78) . Cronin 
claims a figure of 2000 foK^research, evaluation, 
planning and assessment, a somewhat broader category 
than we are considering here. 



(c) Costs by activity type . Stanfield (1981) has 
divided SEA evaluation activities into four main 
groups 1 
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- research analyses , thai is ,^stydies' of an 
essentially ad hoc nature' involving design, 
data collection, analysis ajidLre porting* 
either dealing with some prospective course 
of action or policy issue, or with a 
requ^ed^etrospective assessment ; 

. - information systems , that is , developing 
and operating computerized information 
systems with data on the s'tate educational 
enterprise, schools and students; 

- testing , e.g., statewide testing of a. 
particular grade ; and V 

- expert assistance , or consulting, for a> 
variety of clients, e.g., legislative staff, 
SEA management and LEAs . 

These categories are important later on for 
considering the potential contribution of 
management consulting techniques. 

SEA evaluation internal cost breakdown by activity 
type is estimated as follows t 

research analyses $5.2 million Z$% 

info systems 5.2 25 

testing 5.2 25 

expert assistance 5.2 ^ 25 



$20.8 million 100$ 
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SEA evaluation external (outside contract) cost break 
down by activity type is estimated as follows: ^ 

> 

research analyses $1.68 million 15% 
info systems , O.56 5 

testing 8 A 75 

expert assistance ' O.56 * 5 

• $11.2 million 100$ 

These figures are based upon the* experience of the \ 
respondent s-feates. Although the internal cost 
breakdown is apparently evenly distributed between 
activities, individual figures are -quite varied. 
It appears that research analyses represent a 
highly skewiS category, substanj^al for larger 
states and/or those with major categorical programs,' 
and negligible for many others • There appears to 
. be good agreement on the distribution of external 
'oosts, most going for testing contracts. 

For the re9ord,^.t may be of interest to /Look at the 
figures for internal and external cost combined , 
from which it can be seen that testing is clearly 
the largest category* 

research analyses , . $6/88 million 22% 

info Systems ^ 5-76 • 18 

testing * 13,. 6 # ^2 

expert assistance 5-76 18 

\ . * $32 million 100$ 
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(d) Discretionary funding . w Funding if or. SEA evaluation 
tends to be specific and associated with particular 
programs. Consequently there is little discretion 
in chobsing tasks to work on, pe^h^ps 10% of budget, 
based upon the Experience of the respondent -states . 
.There may even be little discretion .in hpw to work 
on tasks, and^vRat results to provide. 1 This is 
duetto tj^pr^ctice of mancJatifig certain types of 
methodologies and output formats at the time of 4 
f undi/ngf in theWLnterests of data consistency and 
accountability. It is possible, in^additioh, 
ikt the appearance of explicit discretionary 
funding is deliberately avoided by SEA Revaluation 
units' in the present times' of budgetary constraint.' 



2 .5 Individual Evaluation Tasks aftid Costs 

Individual Evaluation tasks vary greatly in size and / 

cost.- This variety is important in considering > 

possible approaches to containing or reducing evaluation 

costs. However, this variety also makes* it difficult 

to describe a typical evaluation, without going into 

long^fcxplanations of other tasks that should also be 

considered. Boruch and Cdrdray (1980) and Frankel 

et al (l979)<>provide information on sizes and costs 

- J: ^ 
of individual tasks. 

• ( '■ ■ 

'At the federal 'level , typical contracted projects last 
2^ months and cost $100-500,000, with some lasting over 
five years, and costing several million dollars. * Case 
study outlines of a Upzen or so projects are given by . 
Boruch^knd Cordray. 



Frankel et al show how the sizes and costs of , 
activities in SEAs and LEAs. overlap. The largest 
urban LEAs have evaluation units ^larger than most 
states. There is an enormous range of activity within 
each,_£j2Ctegory . i/eAs condyct from* zero to over one 
hundred pro jects^each year, with an average of 13 -at 
an average unit cost of about $13,000. SEAs condufet 
the same range of projects each" year, but. with an 
average of 27 at an average unit cost of about $44,000-/ 

While these figures sTiow the expected gradation of 
cost, and hence^pize and duration, from- federal through 
SEA t LEA projects, they are still too general for our 
purpose in this pilot <study. / - x 

Much more helpful is the ^ero Base Budget of the 
. California Office of Program Evaluation and Research 
(California slate Department of Education, 1978). m 
While California is not typical ot other states , it 
may be considered as an Example of a highly regarded, 
•comprehensive evaluation program, which probably 
includes, as subsets, the -activities of many other 
states. 'Of particular interest to the present study 
is^the breakdown of evaluation tasks into eight 
functions, and the explicit budgeting of effort against 
each of these, in the area of what we call research 
analyses above (section 2Ac). These functioAs, p 
together with the average allocated percentage "tof 
effort against each, are shown overleaf t 
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» A. Generating evaluation questions 7& 
and issues 
B. Designing evaluations and 7 
special studies 
_ m C. Developing data collection 9 
procedures *and instruments % * 
D. Collecting data 23 
_ E. Analyzing and interpreting data 17 * 

F. Writing, presenting and 8 
^disseminating reppr'ts 

G. Proyiding* evaluation assistance 21 
to county offices and local 

\ school districts 

H. Providing evaluation assistance ^ 
t * to SEA management and program 

staff _^ 

In the actual budget document, these functions are 

subdivided further. ' The' breakdown above enables us 

to see the relative amount of effort expended on the 

three major phases of the tasks s 
» • 

front end', or set up, activities - lk% 
core data collection and analysis ^9 
back endj or presentation, 33 
application and assistance 
activities ' / 

99% 



4 



MANAGEMENT CONSULTING TECHNIQUES 

'This section considers some aspects of management 
consulting^of relevance to its potential for -reducing 
the costs of SEA evaluation, rather generally at this 
stage in tlie study v 

Traditional* managemejSI consulting is mostly concerned ^ 
with the, internal operations of an organization , e.g., 
the management of staff, organizational structure, 
production, marketing, planning and control (Kubr, 1977) 
In the present case, however, we are considering 
management consulting ' applied to evaluation or 
evaluation-like activities ^ which is a somewhat ™ 
specialized subject. 

As a first step is seems useful to identify four 
related skill areas, namely, management consulting, 
technical consulting, evaluation, and research. These 
are all, 'to some extent, examples of what* might be 
callei Mielping-enquiring skills, and which are often 
difficult to distinguish clearly orie from the other 
because of overlap. These skills share approaches 
of the general formi 

- contact with the client 

* - agreement on what is to be done 

- fact finding 

- fact analysis 

- conclusions/recommendations 

- presentation to the client 

- application/use by the client 

with some degree of liaison/communication with the 
client throughout, j) 2{) 
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One of the ma jor \reasons for confusion lies in the 
difference between a skill area and a practitioner, 
e.g*., between evaluation and an evaluator. Because 
of' legitimate' overlap we^may find, ftora time to time, ^ " 
a management consultant performing research, an 
evaluator performing technical consulting, or a 
researcher performing evaluation. What creates 

\ distinctions in*practipe is the sense of characteristic 
/^skill shown by a person or an organization over a 
period of time, and evidenced in their track record 
and. client references. Consequently, it ma# not be 

p particularly helpful to describe a specific project 
-as a management consulting project, since other 
professionals may feel that it is something that they 
themselves might do with equal confidence. « * 

Having said this, nevertheless, there are differences 
which are* characteristic of the different skill areas, 
and which lead to different classes of ^clients, 
different expectations of' what they can perform, and 
different approaches to tasks in practice. A medical 
, Example may -Bapture some of this characteristic ' 

difference through the illustrative equivalences belowi 

* * * 

- management consultant general practitioner 

technical consultant surgeon 

evaluator * radiologist 
researcher physiologist 



The general practitioner^ typically has the broadest 
scope in helping the cltBnA, and the client is aware 
of this. The surgeon is more specialized, restricted 
mostly to his area of expertise. The radiologist has 
a measuring/diagnostic function, but in regard to a 
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jpdque methodology, \n this case, X-rays • The 
physiologist sees the client more as an object of 
research - his true clients are- his research sponsor 
and his .peers across the country. This last 
distinction will tend to separate out the first three 
with' their emphasis on helping from research with its 
emphasis on enquiring, and its "hidden clients"* 

Another, "hypothetical illustration closer to 
educational evaluation may make these distinctions 
more specific. ♦ Consider a state level committee 
funding an educational program X to achieve purpose Y, 
for example, a pull-out program to* help refugee 
children become more- competent in "basic skills so that 
they can participate in the "basic school* programs . 
The committee might hire a management consultan^ to 
ask if funding should "be continued, with what changes 
if any, or it there is another ("better) way fio achieve 
the goal sought than- "by the program iri^its present form. 
A technical consultant might be hired to advise on 
a new instructional module for the program, and an 
evaluator to measure andf report on the program and its 
outcomes. Finally, a ^researcher might obtain funding 
from other sources • to examine', with the committee's 
approval, sex-linked differences In refugee acceptance 
of the program** While this example may seem a bit ' 
artificial, it does show management consulting as. 
clearly the most directly concerned with the client's 
decisions and options for action. v 

We now turn 'to consider management consulting and 
evaluation more closely.' The steps of the general » 
helping- enquiry approach outlined above offer a 
useful set of headings . 

2, 



\ 

(a) Contact with the client * Management consultants 
usually deal with the client directly, rather 
than solely through staff intermediaries, i.e., 
would meet with the actual committees, not 

just with their staffs, although the latter 
might serve as liaison throughout the project. 

(b) Agreement on what is to be done . The goals of, 
the activity would be negotiated jointly with ( 
the consultant, rather than being fixed by the 

f under in advance, and then}followed without 
« discussion or amendment. * 

(c) Faclf finding . The 'management consultant tries 
only to find out enough to answer the questions 
and ^prove the point, making use of key people 

■ as respondents and the " 80/20" rule to find out 

what is necessary and sufficient as economically 
fc as possible. " (The 80/20 rule says that 80$ of 
1 the effect is due to 20# of the activity or 
resource, and vice versa.) The evaluator on 
the ojther hand is often required to be 
comprehensive . 

(d^ Fact analysis . This is generally not a 

, distinguishing activity. The • evaluator will 
typically use more sophisticated analytical 
techniques than the management consultant, but 
these may not be as robust as the latter's'in 
the face of unreliable data and aA often tight 



schedule . 
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(e) Conclusions/recommendations . The fnanagement 
consultant offers recommendations for change , 



addressed to the client's decision process, 
and with the prior expectation of the client 
that this wiir be so . The evaluator may have 
to be content with publishing the report. 
However, this is often a public document, in 
contrast to the often confidential management 
consulting report. 

(f ) Presentation to the client . Both evaluator 
- and ipanagement consultant present their 

results to the. client.^ The emphasis of the « 
evaluator appears to be on the results them- 
selves, and hqw they are arrived at. The 
management consultant tends to focus on. what 
they mean for the client. 

(g) Application/use bv the client . Both management 
consultant and evaluator are often disappointed 
that the client does little with their work. 
However/ this is generally less the case for 
the management consultant, where client 
expectations and the thrust of the work' lay 

the groundwork for readiness to act upon 
practical and advantageous recommendations. 

(h) Liaison/communication with the client . In 
general, the management consultant invests 
heavily in communication with the client, at 
the outset in negotiating pro ject. goals , during 
the project, and in concluding discussion and 
presentation. This leads to flexible, well- 
controlled work that tracks the client's 
possibly changing interests, and to surprise- 
free final results that are most likely to be 
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persuasive to action. In a, sense the 
consultant becomes a partner, to the client 
through close working together, with inoreased 
perception of client interests and concerns 
throughout the project. *• This is at some 
variance with the confrontational, quality 
control image sometimes put forward of 
evaluators . 





In considering the overall distribution of effort in 
the. course of a project, we may refer back to the 
California summary figures^Jji section 2.5 above In 
contrast, a management c<msulting project might have a 
breakdown of i \^ 

front-end, or set-up, activities 30% 
core data collection and analysis ^0 
back end, or presentation, 30 
application and assistance 

activities 

100% 

In addition, rather than data oollec^ion and analysis 
being treated as a major block of effort expended 
without client contact, the management consulting project 
.may procefed in ari iterative, ad&ptive manner, with 
frequent <\lient contact to discuss and respond to 
emerging^e^ults as they develop. 

In conclusion, therefore, it is clear that management 
consulting and evaluation have a great deal in common, 
and yet, f at the same time, certain characteristic 
differences. Some of these differences lie in the 
approaches themselves, and others in the client 
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environment in which they are practiced. The mors 
specific question of applicability to management 
consulting to (SEA) evaluation is discussed *in the 
next section. 



< 
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DISCUSSION . * 

In this section we discuss -the applicability of 
management consulting to 'SEA evaluation, and eS^Linate 
tlie potential savings that might result, *"\ 

A^f the last section showed, there is a^considerable . 
degree of overlap between management consulting and 
evaluation. Yet thena. aye imp^tant differences, 
including the essential nature of $the product or 
service rendered, the interface [with the client, and 
the audience for the results. 

The ^fsential nature of the management consulting - 
product is advice on imminent decision making, while 
that of evaluation is more typically a multi-purpose 
report or data resource, for use now or later. The 
clierit interface is typically at a higher level in the 
case of management consulting than for evaluation, it 
is serviced more frequently 'throughout a project, 'and 
the management consultant comes to resemble a partner 
of the client rather than a potential adversary. The 
audience for a management consulting- report is usually 
small, confidential and directly addressed > while for 
an evaluation report *it may consist of a whole set of 
parties, each looking for something to support their 
own c^fe . 

Despite the lament about lack of impact, evaluation 
pften starts with good intentions. t For example, the 
functional breakdown of evaluation that is described 
in the California Zero Base Budget (see section 2.5) 
is entirely appropriate. So, also, is the advice of 
Alkin (1980), who *says * ...the most important part of 
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an evaluation is ,f framing the decision context" and 
that activity is more important than anything else 
which the evaluator might do." However, these 
intentions are often not realized. Boruch and Cordray 
quote a UCLA study of over one hundred LEA evaluation 
reports, in which they are rated for the presence oj 
thirteen key elements. The most common are: 

97% Evaluation results are described or 
presented 

92% Data collection sources, such as tests, 
records, or observation' forms , are 
identified 

f 81% Data analysis procedures are described 
or are evident (as in detailed tables) 

The least common arei 

/ 

1095 The reliability of the data collection 
sources, and the validity of the data 
collection sources for the purposes 
intended is described 
17% The program or product or other object 

^under study in the evaluation is described 
so that the form of its actual implement- 
ation is clear 
♦ 

28?5 The congruence of the conclusions with 
the information provided is described 
or evident 

In' general, technical issues are dealt v/ith, but the 
reports* are not effective for other reasons. While 
these are' assessed of LEA reports, it seems likely that 
they are also representative of many SEA reports, too. 



It is, of course', true that some SEA evaluators are 
doing effective 4 management consulting some of the time 
(in the restricted sense used in this repprt) , but this 
is not characteristic of the mass of SEA evaluation 
activity. 

As noted earlier (section 2.4), SEA evaluation consists 
of four different types activity, concerned with 
research analyses, information systems, testing, and 
* expert assistance. This breakdown snakes it convenient" 
to assess the applicability of management consulting 
to SEA evaluation as a^whole in terms of its 
applicability to these/four individual types of 
activity. We will now consider these one at a time, 
assessing potential applicability as - low , medium or high 

Research analyses (in the sense used in 2.^) are 
enquiries to assess the value of.^n existing program - 
or to contribute to the formulati&Vi of policy and 
^policy level decisions. As s^ch^, management consulting 
can play an important^role , provided that the clients 
are willing to allow'the consultant access to'* - \tieir 
decision making process. : If the client primarily 
wants a paper report to serve as a critical record of 
a program's activities, thfen much of the potential 
value of the management consulting approach is lost. 
In this consultant's view; the potential applicability 
of management consulting^to research analyses is high. 
Ironically, it is possible that management- consulting 
approaches may have greater impact than more 
theoretically driven ones, precisely because of their 
client-centered emphasis. 
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Information systems take in data frW 5 various sources, 
e.g., LEA reports, testing scores, research results, 
demographic and financial files. However, they are 
not an area where management consulting appears to 
have much potential impact. This is partly because 
they are enjoying the benefits of falling hardware 
costs on the one hand, and on the other, a part of 
their costs due to programming and data entry are 
not amenable to consulting approaches. The area 
where this may be possible is in the economic 
selection of data that will prove to be influential, 
although this runs counter to present data base trends, 
which tend to be comprehensive. Potential applicability 
of management consulting is judged low . 

Testj^fig is somewhat similar to information systems 
from the point of view of applicability of management 
consulting. Where comprehensive testing is required, 
building up a comprehensive data base of scores, there 
seems little that management consulting can contribute, 
as compared, for example, to technical consulting in 
low cost testing techniques. This is an important 
area of costs to address, nevertheless,, since it 
amounts to the largest proportion of evaluation costs 
overall. Potential applicability of management 
consulting is judged low , but that of technical 
consulting is judged high . . 

Expert assistance, itself, is a* form of consulting, 
but in SEAs this evaluation component usually consists 
of many smail activities. It seems that although 
management consulting is relevant in principle, in 
practice the potential applicability is only medium . 
This is because the key question here is how to manage 
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a large number if small activities so that they do not 
exact a disproportionate toll on the SEA evaluation 
utfit, rather than how ta reduce the individual costs 
of a few larger ^activities- as in the case of research 
analyses. , \ 

Ideally, savings to be associated with the ratings 
above (high, medium, low) should be developed from 
cgtse studies or other detailed enquiry. This has' not 
been possible in the present pilot study. We approach 
the question of savings in two stages: how might they 
be achieved, and how much might they amount to. 



There are three ways in which management consulting can 
lead to savings . The first is in reducing the amount 
of effort allocated to data collection, and analysis. 
The second is by ei>abling the client ik> make practical 
tradeoffs between approaches (with their anticipated 
results) and costs, considered at the level of the 
entire project. These can yield much greater savings 
than tradeoffs at a lower level, e.g., choice of computer 
analysis techniques". » The third is by continually 
refocusing effort on the key questions so that, although 
total project effort is reduced, it is progressively 
concentrated on what emerge as the determining issues. 

Turning now to the question of the possible magnitude 
of such savings, this consultant believes, on tjie basis 
of experience, that indicative levels of savings that 
are potentially achievable aret 

high 30?5 
medium ISfo 
low ' Vl^Ojg 
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These estimates are based on the assumption that 
conditions for management consulting apply, e.g., in 
respedt of access to the client's decision making, 
and in the client's desire for advice rather than just 
data. 

These figures may now be put together with the SEA 
evaluation costs breakdown t of section 2.^, giving 
internal savings of aboulj $2.3 million, and external 
(contract) savings of about $0.6 million, for a total 
potential savyigs of $2.9 million annually. This is 
almost ten percent of gross SEA evaluation costs. 

j It may be argued that these potential savings are not 
very significant, when, for example t the California 
Zero Base Budget exercise requires a savings of 2CP 
to be explicitly identified. However, what is of 
interest is that the ZOfo savings that are in fact/ 
identified only reduce the budget by about 1% in the 
area of the evaluation functions described above,! 
even though , they .make -up 35% of the salary cost. \ /' 
' This suggests .that, at the moment, evaluation in the 
- research analyses sense does not have a lot of slack. 
Consequently, if management consulting were to offer 
a means of reducing costs while maintaining effective 
output's ,. then this would-be particularly advantageous. 

These savings axe no't the only benefits that might 
accrue from the use* of management consulting in SEA 
evaluation. To these direct savings we must add the ■ 
indirect benefits deriving from the leverage that 
these tasks have upon the- state educational' policy 
and management process and upon actual programs, 
resulting from improved, timeliness, better under- 
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standing, and quality of the results. Once the concept 
of leverage is raised, we come to the broader question 
of the optimal level of evaluation expenditure within 
the context of the entire educational enterprise,, 
^athei; tharl the narrower question of how to reduce, 
costs alone of SEA evaluation (see Appendix A), (y 

It should be noted, too, that these savings are only 
potential, and that, even if they are achievable iii 
principle, there are a number of obstacles to their 
realization. These obstacles mainly have to do wrth 
the near universal resistance that greets almost any 
proposals for change. Funding agencies, SEA manage- 
ment, and program managers, have all got used to a 
certain way of dealing with evaluation and evaluators, 
ways of writing specifications, types of results to 
expect, relationships (if any) during the work, and 
so on. ' Evaluation staff may feel either that they 
are doing much of this already, or that it will deskill 
them technically to operate in this mode. LEAs may 
feel that management consulting will lead to meddlina 
in^heir own affairs, unlike the limited impact of 7\ 
current evaluation. 

For these reasons, confident estimates of achievable 
savings cannot be made within this pilot study. These 
must await further research, recommendations on which 
are given in the next section. It seems clear, however, 
that the greatest obstacles lie primarily on the demand 
side , among th^/ various parts of what Alex Law calls 
the "client system" , rather than on the si^pply side,. 

Before leaving the ^lestion of savings, it should be 
noted that arguments such ,as those above can also be 
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applied to federal spending on ^valuation, at least 
60% of which certainly lies in the research analyses 
category; Estimates of potential savings at the 
federal level, then, come to about $^*5 million 
annually. These might .also be explored in the course 
of further research. f ^ 
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RECOMMENDATIONS 



In the light of this pilot studJfT certain recommend- 
ations can be made on further research in 'the area of 
costs . 

(a) It is difficult to tfetke this line of enquiry 

further without dealing with actual evaluations, j\ 
.specifically of the research analyses category, tnat 
can serve as a test for the applicability of / 
management consulting in practice rather than just 
in principle. It is important', therefore to study 
actual evaluations as the next step. Three areas 
that should receive attention are i 

- the wfcty^ija which the evaluation arises, or 
v is negotiated with the client, and the 

client's" expectations (demand side); 

- the actual design aftd/or conduct of a 
number (2-5) of case study evaluations 
using a management consulting approach, 
examining costs, quality of results and 

* timeliness (supply side); and ' 

- th6 response of the client system to the 
evaluation (demand side). 

In addition, it is important to select the evaluation 
contexts carefully, to design a study that is as 
little burdensome as possible to the S^A evaluators 
and their clients, whose cooperation is needed, 
and to consider how the results' of the enquiry - (if 
valuable) can best be dysseminsCted persuasively. 
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(b)'The present study treats a number of topics in a 
preliminary fashion that should be followed up if 
the results obtained! are to "be regarded as -author- 
itative rather than Barely indicative . These 
include i ' f 

- further refinement of the quantitative 
^estimates of SEA evaluation costs; 

- extension of these estimates to federal/ 
LEA levels j 

- extension to consider other means of 
^ reducing costs, e.g., technical consulting 

approaches to testing? and 

' - further review of the costs of. management 
support systems, both to build up a more 
extensive set of reference ratios to* which 
the percentage of program expenditure , 
allocated to evaluation can be. compared, 
and to develop further material on the 
, . concept of optimal levels of evaluation 

w t - * * * • -~ expendilfore • 
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A.aCOSTS of management support services 

The dual questions of how to' reduce costs and what is 
the optimal ainount to spend are not only asked of 
evaluation. TJley are as£ed of a variety of 
organizational^activities , including) data processing 
and management inanimation systems, market research, 
advertising, research and development,, planning, 
^quality control, etc* These activities are in some 
way secondary to the primary business of the 
organization, - ^hetjher- it isimaking cereal, producing 
oil, running a hospital, teaching students, insuring 
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risks, etc. It is because of this sense of being 
secondary, although important, that we refer to them 
as management support services. 

The first thing to establish is that there really are 
two questions, and that they are quite distinct. 
There are lfeast three approaches in general to siJfch 
*costsi 



overhead s they are rolled together wit£ 
telephone costs, miscellaneous supplies, etc., 
as overhead, and assumed to play no -direct - 
leverage role in the main business; Once 
this is dona, there is only one thing to do 
ne3rt,„_- t and tlmt_is to try_io. reduce overhead- 
costs as far as possible, since by assumption' 
this will have no effect on the main business 

of the organization. 

i ' 

optimal \ there is assumed* to be some optimal 
level of expense on these services , -after 
which diminishing returns set in. This 
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optimal level is conceptually approached as 
a top-down abstraction, but is more likely 
to be developed in practice, as the sum of a 
set of bottom-up justifications for optimal 
expenditure on a case by case basis, e.g., by 
program or department. 

prevailing level i the confidence to set a 
given level of expenditure derives from 
prevailing levels among the peer group, 
rather than from beliefs about its optimal 
benefits for the particular organization 
per se . 

There is also, by default, a fourth approach which 
consists of muddling through without any explicit 
^policy on investment level in such services. 

* 

The cost reduction approach to evaluation can easily 
become part of the overhead approach. It is assumed 1 
that if service levels are* maintained, i.e., there is 
no discernible effect on the educational process, then 
^.it is desirable to cut evaluation costs as far as 
possible. That is not an unreasonable point of view, 
although it may not be the most advantageous in the long 
run. It may also be coupled with the prevailing level 
approach "since fhere" is a" certain amount of inter- 
organizational comparison either explicit or implicit 
in the educational world . 

The optimizing approach is quite different, and requires 
that some consideration be explicitly given to the 
leveraga^gfect of evaluation on the whole education 
process ;^Pto the extent that it can be shown that 



* there is more effective use of the social investment 
in educatiQri^tlTrough evaluation, then spending on 
evaluation 'should be increased . This leads diretctly 
to the question of how to show this, and theh to the 
question of what to do if a leverage role cannot be 
shown. The optimizing approach is the most ambitious 
for evaluation, but the stakes are high. If it is 
truly effective, then its role could increase markedly, 
but if not, it may decline correspondingly. 

We have seen above (section 2.3). that evaluation at 

federal, state and local levels combined represents 

about 0.16% of the educational expenditure ,at the 

elementary and secondary level, and that federal levels 

run about seven times higher than the other two. How 

can we judge this in comparison with other areas of 

• f 

activity? 

The brief research possible on this question in the 

course of the pilot study has not been very fruitful, 

•» 

but some results are given below for data processing 

.> 

and management information ^systems . 

The cost reducing approach applied to data processing 
looks at the breakdown of budgets in the same way as 
in section 2 A above. Shaw (1980) gives the breakdown 
in-government data^pcocessing .costs $ 



salaries 


W 1 


hardware 


k6 


supplies 


,9 


communications 


1 


other 


1- 




100% 



Software was not estimated, but Would run in the 5-10% 

* & * 

range. She is also able to give some prevailing levels 
of cost in a number of different Sectors as a proportion 
of total budget or revenue i 

education , l-3# - ' 

metal products * 1% 

health care 1—296 

These figures include operational support such as 
payroll and financial accounting as well as management 
information systems. 

Turning to the optimal approach, there is very little 
^available here. What is , represented is the cost benefit 
approach to the cost of individual information system 
projects, which is a component of the optimal approach 
(King^and Schrems , k 1978) . 

* 

The prevailing level approach is represented by Shaw's 
figur© above, but these are for all of data processing, 
not management information systems. Figures for the 
latter (excluding communications) are given by the 
Diebold Research Program as averaging O.527S of 
corporate revenues across industry (MIS, 1979). , 

\ 

This figure is the one most comparable to evaluation. 
To make the comparison more realistic, it would be \ 
appropriate to separate out the testing component of 
evaluation, since it plays a different role than 
management information - it is more aa product, This 
very crude comparison suggests that evaluation 
expenditure is low compared to prevailing levels on 
, management information systems, in industry, but that 
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these levels are much lower than the three to five 
percent sometimes urged^by evaluators for specific 
programs. * There are, of course, also additional 
cost^ analogous to evaluation, such as market research, 
but these have not been studied at this point. 



In conclusion, therefore, it appears likely that on 
a prevailing le^vel basis evaluation expenditure could 
be increased to the advantage of the overall 
educational enterprise considered as one of the most 
important of contemporary social investments. 
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